Most of the research done on quantum error correction studies an error model in which each qubit is affected by noise, independently of the other qubits. In this paper we study a different noise model -- one in which the noise may be correlated with the qubits it acts upon. We show both positive and negative results. On the one hand, we show controlled-X errors cannot be perfectly corrected, yet can be approximately corrected with sub-constant approximation error. On the other hand, we show that no non-trivial quantum error correcting code can approximately correct controlled phase error with sub-constant approximation error.